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Abstract — In this paper, we consider the problem of optimal 
demand response and energy storage management for a power 
consuming entity. The entity's objective is to find an optimal 
control policy for deciding how much load to consume, how much 
power to purchase from/sell to the power grid, and how to use 
the finite capacity energy storage device and renewable energy, 
to minimize his average cost, being the disutility due to load- 
shedding and cost for purchasing power. Due to the coupling 
effect of the finite size energy storage, such problems are chal- 
lenging and are typically tackled using dynamic programming, 
which is often complex in computation and requires substan- 
tial statistical information of the system dynamics. We instead 
develop a low-complexity algorithm called Demand Response 
with Energy Storage Management (DR-ESM). DR-ESM does 
not require any statistical knowledge of the system dynamics, 
including the renewable energy and the power prices. It only 
requires the entity to solve a small convex optimization program 
with 6 variables and 6 linear constraints every time for decision 
making. We prove that DR-ESM is able to achieve near-optimal 
performance and explicitly compute the required energy storage 
size. 

I. Introduction 

The increasing penetration of renewable energy and dis- 
tributed generation inevitably increase the uncertainty in the 
smart grid. Energy storage reduces the impact of these un- 
certainties by smoothing out fluctuations and reducing the 
mismatch between supply and demand (TJ, 12 . Under such 
circumstances, in order to guarantee reliable delivery of elec- 
tricity to consumers, and to ensure stability of the transmis- 
sion and distribution systems, energy storage technology and 
demand response schemes are both being integrated into the 
power grid. Therefore, it is essential to design control schemes 
that best utilize both approaches. 

In this paper, we consider the problem of optimal en- 
ergy management for power consumers with energy stor- 
ages. Specifically, we consider a power consuming entity, 
e.g., a group of households or a commercial building, who 
is equipped with a finite energy storage device, and needs 
to meet his power demand by using his renewable energy, 
stored power, and purchased power from the grid. The energy 
consuming entity pays the grid for drawing power from it, 
but can also sell power to the grid to compensate some of the 
costs. The objective of the energy consuming entity is to find a 
control policy for deciding how much load to consume, when 
and how much power to draw from the grid, how to charge 
the energy storage, and how to sell power back to the grid, 
so as to minimize its time average cost, i.e., disutility due to 
load shedding and payment to the grid. 
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This problem is a challenging problem. The finite ca- 
pacity of the energy storage device couples all the storage 
charging/discharging actions across time. The power prices 
for purchasing power from the grid is time-varying, which 
results in time-varying costs. Also, the available renewable 
energy is stochastic and may be difficult to forecast long 
beforehand. Moreover, the power consuming entity's disutility 
due to power consumption may also vary over time due to 
the changing system environment, e.g., temperature. Finally, 
the entity's ability to carry out demand response further 
complicates the problem. 

There have been many previous work on developing op- 
timal control schemes for optimally utilizing energy storage 
systems and demand response in the smart grid. 0, 0, 
|5| formulate the problem of storage management using dy- 
namic programming (DP) and derive threshold-based control 
policies. [6 1 develops energy storage control policies using 
receding horizon control. Q, (SI formulate the problem of 
finite horizon storage management as convex programs. Q 
uses quadratic control techniques to study the scaling effect 
of energy storage in the power grid. [ 10 1 considers the problem 
of demand response with energy storage in a finite horizon, 
and formulates the problem as a convex optimization program. 
lITTIl and ifTZl develop optimal multi-stage power procurement 
and demand response schemes that do not include storage. 
However, the aforementioned works either assume that the 
system is static, or require complete statistical knowledge 
of all the system dynamics, or ignore the physical energy 
storage capacity constraints, or do not include energy storage. 
Three recent works |fl3l . Ifl4ll . lfT5l construct low-complexity 
algorithms for energy management based on the Lyapunov 
optimization technique [16|, and explicitly consider the actual 
energy storage control. However, in these works, the energy 
storage devices are used mainly as a tool for storing and 
supplying power for the consumers. Thus, they did not investi- 
gate the potential economic aspects of the energy storage, i.e., 
selling power back to the grid, as well as demand response. 

In this paper, we consider both demand response and 
energy storage management. We explicitly take into account 
the fact that the energy storage has finite capacity and the 
system environment can be time-varying. We develop a light- 
weight energy management scheme called demand response 
with energy storage management (DR-ESM), which does not 
require any statistical knowledge of the system dynamics, and 
only requires the user to solve a simple convex optimization 
problems with 6 variables and 6 linear constraints for deci- 
sion making. We also explicitly compute the required energy 
storage size, and prove that DR-ESM is able to achieve near- 



optimal performance with the chosen energy storage capacity. 

This paper is organized as follows. In Section [TTJ we 
state our system model. In Section [Til] we first develop the 
energy storage management algorithm (ESM) for optimal load 
serving and analyze its performance. We then extend our 
results to include demand response and present the demand 
response with energy storage management algorithm (DR- 
ESM) in Section ITv] We generalize our results to Markovian 
systems in Section |V| Simulation results are presented in 



Section VI We then conclude our paper in Section VII 



II. System Model 

We consider a system where a power consuming entity 
(called user in the following), e.g., a group of residential users, 
or a commercial building, is trying to meet his demand by 
purchasing power from the grid, and is trying to utilize his 
renewable energy and a finite-capacity energy storage device to 
minimize his cost. The system is depicted in Fig.[T] We assume 
that the system operates in slotted time, i.e., t G {0, 1, ...}. 
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Fig. 1. User with energy storage. L(t) is the user load at time 
t, r(t) is the available renewable energy. L(i) = L(t) — r(t) is the 
residual demand. di(t) and d c (t) are the purchased power for serving 
the load and charging the storage, respectively. h a (t) is the power 
sold back to the grid and d s (t) is the power drawn from the storage 
to serve the load. r c (t) is the amount from the excessive renewable 
energy to charge the energy storage. The objective of the user is to 
minimize his cost. 

A. Disutility and load serving 

In every time slot, the user first decides how much load to 
consume. We denote this decision by L(t) and assume < 
L(t) < L max . Depending on the system condition, however, 
certain power consumption level may incur some disutility to 
the user. For instance, suppose the user is operating a heater 
to warm his house. Then, if the outside temperature is low, 
operating the heater at a lower heating level, though being 
less costly, may lead to some discomfort for the user. 

To model such a system-dependent disutility and the fact 
that the system condition may be time-varying, we assume 
that there exists a system state S(t) which captures the 
environmental aspect of the disutility function of the user, 
e.g., temperature. Then, the disutility of the user at time t is 
determined by a general function D(L(t), S(t)). We assume 
that S(t) takes value from some finite set S, and that the 
function D(L(t), S(t)) is convex in L(t) for every system 
state S(t) e S and is known to the user. For instance, one 
example of D(L(t), S(t)) can be: 

D(L(t),S(t)) = /3 s(t) (£ t aSt - Ut)) 2 - (D 



S(t) 

Here £ target denotes the target consumption level and fis{t) 
measures how the disutility increases as the power level 
deviates from the target level. 

We denote r(t) the available renewable energy at time t, 
and define L(t) = L(t) — r{t) to be the residual demand. 
If L(t) > , then it is treated as a normal load. Otherwise 
—L(t) will be the excessive renewable energy, and can be used 
to charge the storage. Q 

The user can serve the residual demand L(t) with two power 
sources: power di(t) purchased from the power grid at a unit- 
power price p(t), and power d s (t) drawn from the energy 
storage at a zero price, Similarly, the user can charge the 
energy storage with two power sources: power d c (t) purchased 
from the grid at price p(t), and power r c (t) obtained from the 
excessive renewable energy. If the user stores enough energy, 
he can also sell some power back to the grid at a unit price 
q(t). We denote the amount sold back by h s {t). 

In practice, there will be physical constraints on each of the 
power components. We model them as follows: 

• There is a capacity limit on how much power the user 
can draw from the grid at any time. We denote it by c gr j<H. 
Hence, in every time slot, we have: 

di(t),d c (t) > 0, di(t) + d c (t) < c grid . (2) 

• There is a maximum charging rate of the energy storage, 
denoted by c C har, so that: 

r c (t)>0, d c (t) + r c {t) < Cchar. (3) 

• There is a maximum discharging rate of the energy 
storage, denoted by Cdj S , so that: 

h s (t) ,d s (t)>0, h s (t) + d s (t) < c dis . (4) 

Besides these physical constraints, we note that at any time, 
the user's action must also ensure the following feasibility 
conditions: 

d,(t) + d s (t) = [£(<)]+, < r c (t) < [-L(t)}+. (5) 

That is, at every time slot, the load must be balanced using the 
power from the grid and the energy storage, and the amount 
of renewable energy used to charge the storage must be no 
more than the excessive renewable energy. 

We assume that the renewable energy is bounded, i.e., 
< r(t) < r max . Then, the residual load is also bounded, 
i.e., — r max < L(t) < L max . We also assume that the buying 
and selling prices are bounded for all time, i.e., < p(t) < 
Pmax,0 < q(t) < q max . 

B. Energy storage dynamics 

Under the charging/discharging actions, we assume that 
the energy storage level evolves according to the following 
dynamics: 

E{t + 1) = E(t) - r] e {d s {t) + h s (t)) + r)i{d c {t) + r c (t)). (6) 

'Note here we assume implicitly that we always use the renewable energy 
to serve the load whenever possible, and we also do not consider directly 
selling the renewable energy to the grid. Our results can easily be extended 
to the case when such direct selling is allowed. 

2 Here we implicitly assume that the cost for operating the storage is zero. 
Our results can be extended to the case when there is cost associated with 
utilizing the storage. 



Here r\ e > 1 and r)i < 1 are the coefficients associated with 
discharging and charging power from the storage. Note that by 
using ([SJ, we first assume that the energy storage device has an 
infinite capacity. Later we will show that under our algorithms, 
the energy queue level remains bounded deterministically 
for all time and the needed storage size can be computed 
explicitly. Hence we only need a finite energy capacity to 
implement the algorithms. 

Note that for the discharging decisions to be feasible, the 
following energy-availability (EA) constraint must be met at 
all time: 



(EA) : E(t)>ri e (d s (t)+h s (t)). 



(7) 



That is, there must be more stored energy than what is 
consumed. 

C. Objective 

Assuming that the (EA) condition is met, the user's instanta- 
neous cost is given by its disutility plus the cost for purchasing 
power from the grid minus its gain from selling power, i.e.,r] 

f(t) 4 D(L(t),S{t))+p(t)[di{t) + d c (t)} - q(t)h s (t). (8) 

The user's objective is to find a control policy for deter- 
mining the load consuming, purchasing/selling, and charg- 
ing/discharging actions, so as to minimize his long term time 
average cost, defined: 

t-i 

/ av ^limsup-^E{/(r)}. (9) 

We call an energy management scheme that ensures (EA) for 
all time a. feasible scheme, and use /g V to denote the infimum 
average cost that any feasible policy can achieve with a finite 
capacity energy storage. 

For ease of presentation, below we first assume that the 
quadruple (p(t),q(t),r(t), S(t)) is i.i.d. over each time slot. 
However, we allow the components to be arbitrarily correlated, 
which is typically the case in practice. Later we will relax the 
i.i.d. assumption and generalize the results to the case when 
(p(t),q(i),r(t),S(t)) is Markovian. Finally, we assume the 
following on the system parameters: 

r?iCgrid > VeLmax- (10) 
Since r\ e is the efficiency for power discharging, we see that 



( 10 1 can be viewed as saying that, if the storage can be charged 



with the full power rate c gr jd, then it can be used to support 
a full load for one time slot. Such a condition can typically 
be satisfied in practice, since c gr id is usually large compared 
to L max . 

D. Discussion of the model 

Our model is similar to those used in |4|, |14|, and lfT31 . 
However, in |4j, the problem is solved using dynamic pro- 
gramming (DP) under various approximations of the charg- 
ing/discharging capacity constraints and without demand re- 
sponse. In [14] and [15|, the storage is only used to support 
the user's load but not as an economic tool for the user. Thus, 

3 Note that our results can also be extended to the case when p(t) is a 
function of d[(t) + d c (t). 



they do not consider the possibility of selling the stored power 
back to the grid as well as demand response. 

In the following, in order to best demonstrate our solution 
approach, we first consider the case when the user does not 
perform demand response, i.e., he only tries to serve the given 
load L(t) every time slot. This is an important scenario which 
models cases when the power consumption level cannot be 
changed. The results derived for this load-serving case will 
demonstrate our main solution idea, and will be extended to 



incorporate demand response in Section IV 



III. Algorithm Design for load serving 

In this section, we first consider the case when L(t) is 
a given stochastic process that satisfies < L(t) < L max , 
and that (p(t),q(t),r(t), S(t),L(t)) is an i.i.d. process. This 
corresponds to the case when the load is non-deferrable and 
has to be balanced in every time slot. In this case, the 
instantaneous cost becomes: 

f(t) 4 p(t)[di(t) + d c (t)] - q(t)h s (t). (11) 

We will solve the following two important problems at the 
same time: (i) How should we size the capacity of the storage 
device according to the system parameters and performance 
requirements? (ii) Given the energy storage capacity, how can 
we optimally control the storage device with low-complexity 
algorithms that adapt quickly to the system dynamics and 
provide performance guarantees? 

These two problems are challenging. Indeed, the main dif- 
ficulty in resolving them is the (EA) constraint which requires 
that no energy underflow happens for all time. This constraint 
couples all the actions across time. Hence, most prior works 
tackle it using DP, which typically requires substantial in- 
formation of the system dynamics, and is computationally 
expensive. We instead construct our algorithm based on the 
Lyapunov optimization approach used in [14| and [15]. The 
main idea of our approach is to "temporarily ignore" the 
(EA) constraint and solve the problem, but then show that 
the constraint is automatically guaranteed by our algorithm. 
We will see that, using this approach, we are able to derive 
very light-weight algorithms that can easily be implemented 
in practice and guarantee near-optimal performance. 

A. The energy storage management algorithm ( ESM) 

To start, we first define two control parameters 8 > and 
e > (values to be specified later), where e is used to control 
the distance between the performance of our algorithm and 
the optimum, and 8 is used to determine the optimal energy 
consumption. We then define a Lyappunov function G(t) = 
\{E(t) — 8) 2 , and define the following one-step Lyapunov 
drift: 

A(t) = E{G(t + 1) - G(t) | E(t)}. (12) 

The following lemma first obtains a basic property of the 
Lyaunov drift. 

Lemma 1: The Lyapunov drift satisfies the following in- 
equality for all time: 

A(t) <B- (E{t) - 6)E{r) e d s (t) + Ve h s (t) (13) 

-Tfcdc(f) - Wcit) | E(t)}, 



where B = ^ [77, 
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Proof: See Appendix A. ■ 
To construct our algorithm, we define V — 1/e, and add to 
both sides of ([l3]l the term VE{f{t) \ E(t)} to get: 

A(t) + VE{f(t) I E(t)} (14) 

< B + VE{p(t)[di(t) + d c (t)} - q(t)h s (t) I E{t)} 

- (E(t) - 6)E{ Ve (d s (t) + K(t)) 

-md c {t) - vMt) I E(t)} 

= B-E{d s (t) Ve (E{t)-6) I E(t)} 

- E{h s (t)[ Ve (E(t) -9) + Vq(t)] I E(t)} 
+ E{d,(t)^p(i) I E(t)} 

+ E{d c (t)[Vp(t) + m {E{t) - 9)] I E(t)} 

+ E{r c {t) m (E(t) - 9) \E{t)}. 
We then replace di(t) ~ [L(t)] + — d s (t) in the above to get: 
A(t) + VE{f(t)\E(t)} (15) 
< B + E{V[L(t)]+p(t) I E(t)} 

- E{h s (t)[ri e {E(t) -9) + Vq(t)] \ E(t)} 

- E{d s (t)[ Ve (E(t) -9) + Vp(t)} I E(t)} 

+ E{d c (t)[V P (t) + Vl (E(t) - 9)] I E{t)} 

+ E{r c {t)i ll {E{t)-9) \E{t)}. 

Our power management algorithm is then constructed us- 
ing the "min-drift" principle of the Lyapunov optimization 
technique [16|: at every time slot, choose a set of feasible 
charging/discharging and buying/selling actions to minimize 



the right-hand-side (RHS) of (15 1. Doing so, we obtain the 
following algorithm: 

Energy Storage Management (ESM): Use an energy 
storage of capacity 9 + ?7iC C har (fi will be specified later). At 
every time t, do: 

1) Observe the energy level E(t), the residual load L(t), 
and the power prices p(t) and q(t). [^Define the follow- 
ing weights: 

e 



W h (t) = r) e (E(t) 

w c (t) = m(E(t) - 



p(t) 



W r (t) : 



0) 



(16) 



2) Chooses h 8 (t), di(t), d s (t), d c (t), and r c (t) to solve the 
following optimization problem subject to Q, <j3j, Q, 
Q. Specifically, we solve: 

h s (t)W h {t) + d s (t)W s {t) (17) 
-d c (t)W c (t) - r c (t)W r (t) 



max : 



s.t. 



di(t) + d s (t) = [L(t)} + , 


(18) 


di(t) + d c (t) < c grid , 


(19) 


d c (t) + r c (t) < c char , 


(20) 


h s (t) + d s (t) < c ais , 


(21) 


r c (t) < [-L(t)]+, 


(22) 


d l (t),d c (t),h a (t),d.(t),r e (t) > 0. 





4 Note here observing the residual load L(t) includes observing the renew- 
able energy r(t). This can be done accurately since we are observing its 
instant value, which is equivalent to performing short time forecast. 



Then, perform the chosen actions. O 
Note that the complexity of ESM is very low: at every time 
slot, the user only has to solve a linear program with 5 
variables and 5 linear constraints {18)-{22). This is a very 



simple task and can easily be done. It also does not require any 
statistical knowledge of the residual load L(t), the renewable 
energy r(i), and the power prices p(t) and q{t). Hence, it can 
easily be implemented in practice. We also note that ESM does 
not explicitly take into account the (EA) constraint. However, 
we will show later that (EA) is automatically ensured by the 
ESM algorithm. 

B. Performance analysis of ESM 

The following theorem summarizes the performance of 
ESM. In the theorem, the parameter 9 is defined: 

A maxfpmax, 9max] 



r] e min^max^dis] 



(23) 



Theorem 1: Suppose 9 is chosen according to (23 1, and 
< E(0) < 9 + rjiCcbzr. Then, under ESM, we have: 

< E(t) <9 + T] t c chari Vt, 



fl SM <K v + Be. 



Here is the average cost achieved by ESM, f£ v 

^ c char] 



optimal time average cost, and B = \ [r] t 
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(24) 
(25) 

is the 

O 

Note that ( [24] > is very important. It shows that the energy 
level under ESM will never be negative and is deterministi- 
cally upper bounded, and provides the explicitly bound. This 
shows that the (EA) condition is ensured under ESM, and 
allows us to conveniently size our storage capacity and imple- 
ment the algorithm with a finite capacity energy storage. We 
also emphasize that p4| ) is indeed a sample path result. Thus 
it holds under arbitrary (L(t),p(t), q(t),r(t), S(t)) processes. 
This shows that our algorithm is indeed applicable under more 
general system dynamics. 



Below we present the proof of ( 24 1. The proof of ( 25 1 will 
be given in Appendix B. 

Proof: (Theorem fl) Le t d\ (t) , rf* (t) , d* s (t) , h* s (t) , r* (t) be 
an optimal solution of (|17|i. 



We first prove the upper bound of ( 24 1 using induction. First 
we see that it holds for time t = 0. Now assume the upper 
bound holds at time t. 

1) Suppose E(t) < 9. Then, since d c (t) + r c (t) < c c u ar , 
we see that E(t + 1) < 9 + rjiC c ^ ar 

2) Now suppose E(t) > 9. Then we have W c (t), W r (t) > 
0. Thus, we must have d* (t) = and r* (t) = 0. That is, 
once E(t) > 9, the energy level will not further increase. 
Hence, E(t + 1) < E(t) <9 + ^c char . 

Now we prove the lower bound using induction. Assume 
the lower bound holds at time t. 

1) Suppose E(t) > rj e min[L max , Cdi S ]- Since the maximum 
amount that can be discharged from the storage is 
r^e min[L max , Cdis], we see that E(t + 1) > 0. 

2) Suppose E(t) < rj e min[L max . c<jj S ]. In this case, we see 
from |23]l that: 

max[p max ,g max ] 



r le (E(t)-6),r U (E(t)-9) < 



(26) 



Using ( p~6] >, we see that: 

W h (t),W s (t),W c (t),W r (t) <0. 



(27) 



In this case, we first see from (21 1 that = 0, i.e., 



no energy is sold back to the grid. 
Then, we see that either (i) d* c (t) + r*(t) = c char 
or (ii) d*(t) + d* c (t) = c gr ici ■ This is so because if 
none of them happens, then we can increase d*(t) to 
further increase the value of the objective function of 



( 17 1, which contradicts the fact that d*(t) is the optimal 
solution. 

Suppose now d*(t) + r*(t) = c C h ar - Then, from the 
objective function ( fT7| ) and the constraints ( fl"8) and (19) , 
we see that c? ; * (t) should be as large as possible, so that 



we can use a minimum d* s (t) to maximize ( 17 i. Hence, 
we have: 



df(t) 



[Cg M -d* c (t),[L(t)]- 



(28) 



This implies that either d\ (t) + d* c (t) — Cgrid or d\ (t) = 
[L(t)] + . If ^(i) = [L(t)] + , then d*(t) = 0. Since we 
also have h* s {t) = and d* c (t),r*(t) > 0, we see that: 

E(t + 1) = E{t) - r, e (d* s (t) + h* s (t)) + Vi (d* c (t) + r* c (t)) 

> E(t). 

Now suppose instead we have d^(t) + d*(t) = Cg^. 
Using the facts that djj'(i) + d* s (t) = [L(t)}+ and 
[!/(£)]+ < Lmax, we have: 

E(t + 1) = E(t) - Ve (d* s (t) + h* s (t)) + m {d* c (t) + r*(t)) 

= E(t) - r h ([L(t)}+ - df (t)) + m(d* c (t) + r* c (t)) 

> E(t) - Ve L max + m{di{t) + d* c (t)) 
= E(t) - r) e L max + r?iCgrid 

> E(t), (29) 

where the last step follows from ( |10) . This thus means 
that whenever E(t) < r\ e min[L max , ca, s ], the energy 
level will not decrease. Hence E(t + 1) > E{t) > 0. 
This completes the proof of ( f24) . ■ 
From the above proof, we notice that whenever E{t) > 8, 
ESM will set r c (t) = 0. This is an important feature in that it 
guarantees the boundedness of the stored energy level. Hence, 
ESM can be implemented with finite energy storage devices. 
The reason why ESM can afford not to charge the energy 
storage with the free renewable energy is because under the 
chosen capacity and ESM, the loss of optimality will not be 
more than O(e) even if we occasionally waste a small fraction 
of the renewable energy. In practice, we can modify ESM 
to always store excessive renewable energy, and control the 
system with a "virtual" control sequence that tracks the energy 
level under the original ESM algorithm. 
IV. Energy management with demand response 

In this section, we extend our results to the case when the 
user also performs demand response. Specifically, in every 
time slot, the user also chooses his power consumption level 
< L(t) < L max , and is willing to change his consumption 
according to the system condition. Recall that in this case, the 
user's instant cost becomes: 

f(t) ^ D(L(t),S(t))+p{t)[di(t) + d c (t)} - q(t)h s (t), (30) 



where D(L(t), S(t)) is the disutility experienced by the user 
for consuming a power level Lit). 

A. ESM with demand response 

Here we construct our algorithm to incorporate demand 
response. Using a similar argument as before, we have: 

A(t) + VE{f(t) | E(t)} 

<B- E{h s (t)[ Ve (E(t) -9) + Vq(t)} \ E(t)} 

+ ¥,{d l {t)Vp{t) | E{t)} 

- E{d s (t)r) e {E{t) - 6) | E(t)} 

+ E{d c (t)[r)i(E(t) -6) + V P (t)} | E(t)} 

+ E{r c (t)r h (E{t)-6) \ E(t)} 

+ E{VD(L(t),S(t)) | E{t)}. 

Replacing d s (t) = [L(t)] + — di(t), and rearranging the terms, 
we have: 

A(t) + VE{f(t)\E(t)} 

<B- E{h s (t)[ Ve (E(t) -9) + Vq(t)\ | E(t)} 

+ E{di(t)\Vp(t) + Ve(E(t) - 9)} | E(t)} 

+ E{dc(t)[m(E(t) ~9) + Vp(t)} | E(t)} 

+ E{r c (t) Vl (E(t) - 9) | E(t)} 

+ E{VD(L(t),S(t)) - Ve (E(t) - 9)[L(t) - r(t)] + \ E(t)}. 

We now similarly construct our algorithm by choosing actions 
to minimize the RHS of the drift inequality at every time slot. 
Doing so, we obtain the following algorithm. 

Demand Response with ESM (DR-ESM): Use an energy 
storage of capacity 9 + ^jCchar- At every time t, do: 

1) Observe the energy level E(t), the renewable energy 
r(t) and the prices p{t) and q(t). Define the following 
weights: 



w h (t) = Ve (E(t)-e) 



p(t) 



Wi{t) 

W r (t) 



Ve(E(t) 

Vi(E(t) - 



Pit) 



(31) 



(32) 

r c {t)W r {t) 



W c {t)=Vi{E(t)-6) 

W D (t)= Ve (E(t)-9). 

2) Chooses L(t), h s (t), di(t), d s (t), d c (t) and r c (t) to 
solve: 

min : V D{L(t), S{t)) - W D {t)[L{t) - r(i)] + 
-h s {t)W h {t) + d t (t)Wi{t) + d c (t)W c (t) 
s.t. (§,(§, (§,(§,0<L(i) <W- 
Then, perform the chosen actions. O 
Notice that even with demand response, the DR-ESM al- 
gorithm still does not require any statistical knowledge of 
the system variables, and only requires the user to solve a 
simple convex optimization program with 6 variables and 6 
constraints, which can be solved very efficiently. 

The performance of DR-ESM is summarized in the follow- 
ing theorem. 

Suppose 



Theorem 2: 

< E(0) < 9 



is chosen according to ( 23 1, and 
Vi c char- Then, under DR-ESM, we have: 

0<E(t)<9 + r ll c char ,yt, (33) 

/lv DR < /av + Be. (34) 



Here /a V DR is the time average cost achieved by DR-ESM, 
and J B = i[7 ? 2 c 2 s+r? 2 c 2 har ]. o 

Proof: See Appendix B. ■ 

V. ESM with Markovian system dynamics 

Here we extend the our results to the case when the system 
parameters, i.e., the quadruple (p(t), q(t),r(t), S(t)) evolves 
according to a finite state irreducible and aperiodic Markov 
chain. In this case, the following theorem shows that DR- 
ESM (and similarly ESM) still achieves similar performance 
as in the i.i.d. case. 

Theorem 3: Suppose (p(t),q(t),r(t), S(t)) evolves ac- 
cording to a finite state irreducible and aperiodic Markov 
chain. Also suppose 6 is chosen according to ( |23] l and < 
E(0) < 8 + rjiC C h ar . Then, under DR-ESM, we have: 

0< E(t) < 9 + Vl c char , Vt, (35) 

0(e). (36) 

Here /| v is the optimal time average cost. O 



rE-DR < f* 

Jav — Jav 



Proof: The result ( 35 1 follows directly from Theorem 



[T] and Theorem [2] because it is a sample path result. (36i 
can be proven using the variable multi-slot drift argument 
development in ifTTl . The details are omitted for brevity. ■ 

VI. Simulation 

Here we present simulation results. For simplicity, we 
only simulate DR-ESM. We assume that the load consuming 
entity's consumption level is equivalent to 10 residential users, 
e.g., the entity is an apartment building, or a small commercial 
building. Every time slot is assumed to be one hour. 

We generate time-varying hourly buying prices by assuming 
that the daily average hourly price is 12^/kWh (according 
to PG&E's residential electricity report |18|) and that the 
distribution is uniform over the annual average day-ahead 
hourly price of PG&E in the year 2010 (data from the CAISO 
daily report on the Federal Energy Regulatory Commission 
(FERC) website |fl9l ). The values are shown in Fig. [2] We 
then assume that the selling price at every time is the same as 
the buying price. 




Fig. 2. The price distribution and the renewable energy pattern 

We assume that the user has a wind turbine with a power 
capacity of 9kW, i.e., r max = 9.[^]We similarly assume that the 



wind power has a uniform distribution over the values shown 
in the curve in Fig. [2j which is obtained by averaging and 
normalizing the 2006 annual recorded wind power of 100 wind 
turbines located near 40.42N, 124.39VF at the west coast in 
California, which has an aggregate capacity of 300MW (data 
from the National Renewable Energy Laboratory (NREL) 
website j20l ). and has a mean value 8kW. We assume that 
rj e = 1.25 and r/i = 0.8, corresponding to 80% efficiency 
for both charging and discharging for the storage, and that 
c grid = 20kW, c char = c dis = 12kW. 

We assume that S(t) takes two values "H=High" and 
"L=Low" with equal probabilities, e.g., representing high or 
low temperature, and that the disutility function is of the form 
given in ([T]i with j3jj — Pl = 1, i.e., 

(37) 



D(L(t),S(t)) = (L^ et ~L(t)Y 1 

with Lg rgeX = 12kW, and target = 8kW - We assume that 
Lmax — 12kW. ^\ These numbers are chosen based on the 
overall 2009 California residential average power consump- 
tion, which is 18.6kWh per day per user, or 0.8kWh per hour 
per user EH . To make the units consistent, we assume that 
the utility is also measured in units of cents. 

According to (23 I, we use a storage of size 8 + rjiC c u ar = 
18V + 24.6kWh. Below we simulate V = {2, 5, 10, 20, 50}. 
Our simulation is done with CVX |22|, and each simulation is 
run for 10 4 slots. For comparison purposes, we also simulate 
the case when there is no energy storage. In this case, we 
assume that the user at every time slot chooses the load 
according to the following Greedy scheme: 

L(t) G axgvmiD(L(t),S{t)) +p(t)(L(t) - r(t)) + . (38) 

That is, the user is trying to minimize the instant cost every 
time slot. 



- With storage and DR-ESM 

- Greedy without storage 



Fig. 3. Average cost under DR-ESM and without storage. 

Fig.[3]shows the average costs under DR-ESM and Greedy. 
We see that DR-ESM is able to reduce the average cost by 
64%- 136%. For instance, when V = 5, DR-ESM reduces the 
average cost from 8.24^ down to —1.61^, which corresponds 
to a saving of 120%. The reason that the reduction can exceed 
100% is because with energy storage and DR-ESM, the user 
can actually make profit by carefully buying and selling power. 
We also note that such significant saving is achieved with a 



5 Since the entity roughly represents 10 household users, this capacity can 
represent the total capacity of several home wind turbines. 



s Since we assume each slot is one hour, expressing the consumption level 
in units of kW is the same as in kWh. 



moderate storage capacity. For instance, when V = 5, the 
provisioned storage capacity is 118.35kWh. Considering the 
fact that the user represents roughly 10 residential users, this 
only requires each individual user to have a battery of size 
12kWh. 

Fig. [4] also shows a sample path energy level process under 
DR-ESM with V = 5 in time slots [101,400]. We note that 
the energy level is always below the provisioned capacity and 
never drops below 0. One interesting observation is that we 
can even implement DR-ESM with a storage of size 75kWh. 
This indicates that our algorithm can likely be implemented 
with much smaller storages. 



- Energy Level under DR-ESM 

- Provisioned Storage Capacity 



Time 101:400 



< B - (E(t) - 9)[ve(d s (t)) + h s (t)) - m {d c {t) + r c (f))]. 

Taking expectations over the randomness of the actions con- 
ditioning in 15(f), and using the definition of A(f), we prove 
Lemma [JJ ■ 

Appendix B - Proof of ( |25| ) of Theorem[T] 

In this subsection we prove (25 i of Theorem[TJ To do so, we 
first have the following theorem, which can be proven using 
a similar augment as in |23|. 

Theorem 4: There exists a stationary and randomized en- 
ergy management policy IT that achieves the following: 

E{/ n (*)} = fa, (41) 

E{%(d?(*) +/»?(«))+%(«?(<) +r?(i))} =0. (42) 

Here the expectation is taken over the random system dynam- 
ics and the potential randomness of the charing/discharging 
and purchasing/selling actions. □ 

We remark that Theorem |4] holds for both the load-serving 
case as well as the demand response case. Note that although 
Theorem |4] shows that there exists such an optimal policy II, it 
may not be implementable in practice. This is because finding 
it requires knowing all the statistical knowledge of the system 
parameters, including the prices, the loads, and the renewable 
energy, and the required energy storage size is very difficult 
to compute (may also be infinite). In this case, our ESM 
algorithm provides a low-complexity alternative to achieve a 
similar performance. 

We now use Theorem [4] to prove ( 25 1 of Theorem [JJ 
Proof: ( p5j ) of Theorem [TJ We recall ( fT~4] > as follows: 

A(f) + VE{f(t) I E(t)} (43) 

< B + VE{p(t)[di(t) + d c (t)} - q(t)h s (t) I E(t)} 

- (E{t) - 9)E{ Ve (d s {t) + h s (t)) 

-Vid c {t) - Wc(t) I E(t)}. 

Now since ESM is chosen to minimize the RHS of ( |43| ), the 
value of the RHS is no larger than that under the policy n. 



Thus, (43 1 holds when we plug in the actions chosen by policy 



Fig. 4. A sample path energy level process under DR-ESM with V = 5. 

VII. Conclusion 

In this paper, we developed optimal energy management 
and demand response schemes for general power consuming 
systems with finite energy storage and renewable energy. 
Based on the Lyapunov optimization technique, we developed 
two very light-weight energy management schemes ESM and 
DR-ESM for load-serving and demand-response, respectively. 
Both schemes only require the user to solve a simple convex 
optimization program for decision making, and allow us 
to explicitly compute the required energy storage size. We 
proved that both schemes are able to achieve near-optimal 
performance. 

Appendix A - Proof of LemmaQ] 

Here we prove Lemma [TJ 

Proof: (Lemma [TJ Using (|6j, we see that: 

E(t+l) -9 = E(t) -9 (39) 

-r]e(d s (t)) + h s (t)) + r)i(d c (t) + r c (t)). 

Square both sides of the above, we get: 

(E(t + 1) - 9f (40) 

= (E(t) - 9f + { Ve (d s (t)) + h s (t)) - Vl (d c (t) + r c (t))} 2 

-2(E(t) - 6){r) e (d s {t)) + h s (t)) - m (d c {t) + r c (t))]. 

Now using ([3]) and Q, we see that: 

[r) e (d s (t)) + h s (t)) ~ Vi(d c (t) + r c {t))] 2 < + r, 2 c 2 char 

Thus, by defining B = |[^^ ta + Vi4hJ< multiplying both 1 y^r /ESM( t U < y» + ~ + 

sides of (40 1 by |, and using the above, we get: ' z — ' v 1 

l -[(E(t + l)-9) 2 -{E{t)-9) 2 ] 



II. This yields: 

A(t) + VE{f ESM (t) I E(t)} (44) 

< B + VE{p(t)[dY(t) + d?(t)\ - q(t)h?(t) I 15(f)} 

-(E(t)-6)E{r, e (d*(t)+hZ(t)) 

-r)id£(t) - »hr?(t) I E(t)} 

= B + Vf* v . (45) 

Here in the last step we have used ( pT) and ( |42| in Theorem 
|4] Therefore, taking an expectation over E{t) on both sides, 
and summing the above over f = 0, T — 1, we get: 



T-l 



E{G(T) - G(0)} + 22 VE{f ESM {t)} <TB + TVf* v . (46) 
t=o 

Rearranging the terms and dividing both sides by TV, we see 
that: 

T-l 

/ ; - : . ! ( ri 11 i : 

(47) 



t=o 



B + E{G(0)} 
V 



VT 



Taking a limsup as T — >• 
E{G(0)} < 00, we prove (p5i 



00 and using the fact that 



Appendix - Proof of Theorem__ 

In this section, we prove Theorem [2] The proof is very 
similar to the one of Theorem [2] except for the lower bound 
of (33 I. Hence, we only present the proof of this part. 

Proof: (Theorem U Let L*(t\ tf(t), d*(t), d* s (t), h*(t), 
r* c {t) be an optimal solution of (32 1. We first notice that L*(t) 
will not directly affect the other actions. Hence, we can treat 
L*(t) as a given load. 

Now we prove the lower bound using induction. Assume 
the lower bound holds at time t. 

1) Suppose E(t) > rj e min[L max , Since the maximum 
amount that can be discharged from the storage is 
7j e min[L ma x, Cdi S ], we see that E(t + 1) > 0. 

2) Suppose E(t) < ?y e min[L max , Cdis]. In this case, we see 
from d23l that: 



< - 



r, e (E(t) - e), m (E(t) - 
We similarly see from ([26]) that 



max[p m ax, <?maxj 



(48) 



W h (t), Wi{t),W c {t),W r (t) < 0. (49) 

Note here we have Wi(t) instead of W s (t). In this case, 
we similarly see that h* s (t) = 0, and that either (i) 
d* c (t) + K{t) = c char or (ii) d*(t) + d*(t) = c grid . 
Suppose d*(t) + d* c {t) = Cgnd, then using (29i, we 



see that E(t + 1) > E(t) > 0. Else suppose d* c (t) + 
r* (t) = c C har- From the objective function ( 17 1 and the 



constraints ( 18 1 and ( 19 1, we again see that df (t) should 
be as large as possible since Wi{t) < in $7$. Hence, 
we have: 

df(t) = min[c grid - d* c (t), [L(t)}+]. (50) 

Using the proof of Theorem [T] we see that this implies 

E(t + 1) > E(t) > 0. 
The rest of the proof follows similarly as in the proof of 
Theorem Q] ■ 
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